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Detailed Action 

1. This office action is in response to Amendment and Declaration under 37 C.F.R. 1.131 
filed on 01/17/02, where claims 1-6, 8-25, 27-33 and 35-^ remain pending. 

Claim Rejections - 35 JJSC §103 

2. Quotation of 35 U.S.C. §103(a) which forms the basis for all obviousness rejections set 
forth in this Office action may be found in previous office action: 

3. Claims 1 and 3-5 are rejected under 35 U.S.C. 103(a) as being unpatentable over Hou et. 
al (Hou) U.S. Patent No. 5,838,313 in view of SidanaU.S. Patent No. 6,081,829. 

Regarding claims 1 , Hou teaches substantial features of the invention as claimed; teaching a 
system comprising: 

a client computer (col 6/lines 38-51) to playback multimedia content (col 8/lines 1-15, col 
record/playback annotations, 6/lines 30-37) and annotations coiresponding to different segments 
(temporal portions) of the multimedia content (e.g. voice segments, col 8/lines 61-63, playback 
marked segments, col 9/lines 9-17); 

maintaining an annotation database (33) having a plurality of annotations coiresponding 
to the multimedia content (col 6/lines 10-21, 38-41, multimedia storage/database), 

provide the plurality of annotations to said client computer for playback (Figs. 6, and 12 
playback segment, col 5/lines 18-25, col 8/lines 43-48); 

send electronic mail messages (send via email col 3/lines 8-13, including multimedia, col 
2/lines 63-64, annotation, col 1/lines 8-10, attachment, col 1/lines 31-39, email create or reply 
means: col 2/lines 25-37, sending email: col 1 1/lines 22-25, generating an e-mail message: col 
6/lines 14-16) including annotations (col 1 1/lines 3-25, Fig. 1) to recipients identified by the 
client computer (mail listing col 4/lines 26-31, Fig. 4), via interface means to receive input data 
regarding new annotation coiresponding to media content (col 6/lines 26-51); 
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opening and loading content data of an electronic mail message displaying located 
content data in the electronic mail message received from the mailbox (retrieved received 
message fi-om mailbox and loaded content including multimedia data and annotation data 
rendered on display, col 4/lines 10-14, use an annotation to reply, col 9/lines 14-24), received 
message not in reply to a previous electronic mail message (col 2/lines 25-30, col 3/lines 13-15); 

generate new annotation that include display located content data in the electronic mail 
message (e.g. based on existent annotation data displayed col 6/lines 21-40); 

add the new annotations (attached media types (26) col 6/lines 10-25, add dynamic 
annotations (39), e.g. voice, col 6/lines 26-37) and save in annotation database (col 4/lines 21-25, 
save repoit col 2/lines 50-51, report includes multimedia as well as attachments, col 1/lines 39- 
44, add the new annotations, col 9/lines 15-32); however Hou does not explicitly teach an 
annotation server coupled to a client computer; 

Sidana teaches a system/method for enabling client using an user interface to be 
presented with annotations, and maintaining a annotation multimedia server computer coupled to 
the client for streaming the multimedia to the client computer (abstract, col 2/lines 34-59) via a 
network (col 4/lines 14-33, Fig. 1, elements 106, 110, 130, annotation server 120), where an 
identifier, identifiers the media content associated with new annotation content (col 8/lines 5-12); 

It would have been obvious to one ordinary skilled in the art at the time the invention was 
made to modify Hou's system with means where an annotation server, coupled to the client 
computer to maintain an annotation database and to manage streaming the multimedia content to 
the client computer, as taught by Sidana, motivation would be enable the user via multimedia 
content identifiers to select for viewing or not a annotation included in a document, access a 
media server to obtain multimedia content coiTCsponding to the content identifier which may 
further be presented in hierarchical for, where and annotation set identifier one or more sets of 
annotations. 

Regarding claim 3, client computer coupled to the annotation server via a network (Sidana: col 
8/lines 5-12). 



Application/Control Number: 09/396,701 (GUPTA et. al.) 
Art Unit: 2152 



3 



Regai'ding claim 4, present an electronic mail message including a multimedia content identifier 
to a user (Hou: annotation event-related data in annotation file, col 7/lines 1-3, event-related data 
including time-slice events, col 7/lines 4-6, time-line mai'kers include markers to specify 
annotation segments, col 8/lines 43-48, 61-63, col 9/lines 8-13); 

receive a user selection of the multimedia content identifier, access a media server to 
obtain the multimedia content (Sidana: col 7/lines 62-col 8/line 26, Hou: col 11/lines 26-34, 
playback the multimedia content to the user). 

Regarding claim 5, transmitting to the media server, an identifier of a temporal segment of the 
multimedia content, and wherein the media server is to provide (stream) to the client computer 
the multimedia content beginning with the identified temporal segment (Hou: col 6/lines 61-66, 
col 7/lines 1-3, col 8/line 61-63); 

wherein said identifier of a temporal segment of the multimedia content is after the 
beginning of the multimedia content, such as time temporal range denoting the a segment of time 
(i.e. beginning/end), temporal range of information denoting beginning and end of a segmented 
multimedia content (e.g. load/display, time, begin/end of objects) (Hou: Figs. 6, 14-15, col 
7/lines 4-51, col 2/lines 52-61, col 6/lines 10-14). 

4. Claims 31-32, 37-39, 2, 6, and 8-12 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Hou et. al. (Hou) U.S. Patent No. 5,838,313 in view of Sidana U.S. Patent No. 
6,081,829 in view fiiither view of Biirel et. al. (Birell) U.S. Patent No. 6,009,462. 

Regarding claim 31, ftiither teach wherein a method comprising: 

receiving an electronic mail message including an annotation coiTesponding to media 
content (Hou: col 4/lines 10-14, media content including multimedia annotation, col 1/lines 39- 
44, col 3/lines 9-13); 

replying to the electronic mail message to generate a new annotation coiresponding to the 
media content (Hou: add or use existing annotation reply, col 9/lines 14-24, col 6/lines 26-37. 
reply to original annotation, col 2/lines 25-30); 
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including in reply electronic mail message multimedia content annotation file (Hou: 
annotation file including event data, col 7/lines 1-3, event data including categoiy, e.g. time- 
based annotation, col 7/lines 4-6, event data e.g. markers or identifier to specify annotation 
segments, col 8/lines 43-48, col 9/lines 8-13); 

displaying a recipient list coiresponding to the identified annotation categoiy in the 
received electronic mail message (Hou: reply by inserting annotation in received message, col 
2/line 25-31, adding annotation, col 6/lines 26-37); 

however the above teaching do not explicitly teach receiving notification that an 
electronic mail message has been received, received electronic mail notifying an annotation 
coiresponding to media content. 

BiiTcl teaches a system/method related the composition and distribution of multimedia 
email, teaching receiving announcement, notice, indication, hint, i.e. notification that an 
electronic mail message has been received (Biirel: col 11/lines 43-52), received electronic mail 
notifying an annotation coiresponding to media content (Biirel: col 12/lines 1-11, audio type, 
col 12/lines 4-44), analyzing an electronic mail message to locate multimedia data and include 
located data in a storage (Biirel: col 12/lines 61 -col 13/line 7) and including an identifier of 
media content to which the annotation content coiresponds. 

It would have been obvious to one ordinary skilled in the art at the time the invention was 
made to include means for receiving notification that an electronic mail message has been 
received, received electronic mail notifying to include an identifier of the media content but does 
not include the media content, as taught Biirell, motivation would be motivation would be enable 
a plurality of client machines to create/receive annotations within electronic mail messages to 
include multiple media content types such, HTML pages of embedded MIME fi'agments all 
implemented with standard existing software, annotate electronic mail messages with mutable 
identifiers for categorized storage/retrieval and access mail via low-bandwidth network 
connections by generating user-linkable annotations within an electronic mail message, as taught 
by Biirel. 
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Regarding claim 2, a media server to manage streaming the multimedia content to the client 
computer, including being communicated to the client upon request (i.e. as needed) (Biirel: col 
12/line 59-col 13/line 7). 

Regarding claim 32, electronic mail notification including an identifier of the annotation (Biirel: 
notification of received/unread labels, col 1 1/lines 43-52, identifier (hot-link) of the annotation, 
col 12/lines 1-11); and including the identifier of the annotation in the reply (Hou: col 5/lines 10- 
28). 

Regarding claim 37, this is the computer-readable media having stored thereon a computer 
progi'am that when executed by processor(s) perfoims the functions disclosed on the system of 
claim 1, rejected for obviousness under U.S.C. 103, this same rationale is also applied to 
software implementation claim, claimed in terms of function, property or characteristic. 

Regarding claim 38, substantially the same as claim 1, and further creating a new annotation 
based on the extracted annotation content and the annotation identification information (Hou: 
e.g. based on existent annotation displayed col 6/lines 21-40); 

add the new annotations (Hou: attached media types (26) col 6/lines 10-25, add dynamic 
annotations (39), e.g. voice, col 6/lines 26-37) and save in annotation database (Hou: col 4/lines 
21-25, save report col 2/lines 50-51, report includes multimedia as well as attachments, col 
1/lines 39-44). 

Regarding claim 39, further teach an identifier of media content to which the annotation content 
coiresponds and an identifier of media content to which the annotation content of a temporal 
segment (Biirel: col 12/lines 61 -col 13/line 7, identifier (hot-link) of media content to which the 
annotation content coiresponds, Hou: identifier of media content to which the annotation of 
temporal segment coiresponds, 6/lines 4-25, annotation event-related included the annotation 
file, col 7/lines 1-3, recording generates time-slice events, col 7/lines 4-6, time-lime identifiers 
(markers) include markers to specify annotation segments, col 8/lines 43-48, 61-63, col 9/lines 8- 
13); and 
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an identifier of an annotation set that a new annotation including the extracted annotation 
content is to be part of (Biirel: col 12/lines 61 -col 13/line 7), wherein the annotation set is one of 
one or more annotation sets coiresponding to categories format types for annotations (Hou: col 
7/lines 4-51, col 2/lines 52-61, col 6/lines 10-14). 

Regarding claims 6, displaying to the user, a user interface (Hou: Fig. 5, col 2/lines 62-col 3/line 
14, canvas, col 4/lines 32-col 5/lines) allowing the user to create a new annotation (Hou: col 
2/lines 52-56, adding/generating means: col 6/lines 10-37) coiresponding to media content (Hou: 
Fig. 5); 

including, as part of the user interface (Hou: abstract: col 1/lines 65-col 2/line 5), a field 
via which the user can identify a recipient that is to receive an electronic mail notification of the 
new annotation (Hou: col 4/lines 26-32 list recipients, Figs. 4, 12, Sidana: col 1/lines 55-62), and 

transmitting the new annotation to an annotation server (Hou: add the new annotations 
attached media types (26) col 6/lines 10-25, add dynamic annotations (39), e.g. voice, col 6/lines 
26-37 and save in annotation database col 4/lines 21-25, save report col 2/lines 50-51), 

wherein the electronic mail message includes an annotation and an identifier of the media 
content but does not include the media content (Biirel: col 12/lines 61 -col 13/line 7). 

Claim 7 cancelled. 

Regarding claim 8, including in the electronic mail message an indication of one of a plurality of 
annotation sets, the one annotation set coiresponding to the media content of the new annotation; 
(Hou: col 6/lines 10-14, 26-51, 61-66, sending/displaying mail: col 1 1/lines 22-25, wherein the 
annotations sets coiresponds to categories for annotations; different types of media, allows the 
user to record/playback annotations, an use can create reports by including multiple media data; 
col 2/line 50-col 3/line 13). 

Regarding claim 9, allowing the user to identify, via the field of the user interface, a plurality of 
recipients to receive the electronic mail notification, as discussed above, and fomarding, to each 
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of the plurality of recipients, the electronic mail notification of the new annotation (Hou: col 
4/lines 26-31, Fig. 4). 

Regarding claim 10, allowing the user to input, via the field of the user interface, a group 
identifier, as discussed above, Hou: col 4/lines 26-31); identifying an electronic mail address for 
each of a plurality of recipients coiresponding to the group identifier; and fomarding, to each of 
the plurality of recipients (Hou: col 4/lines 26-31, Fig. 4), the electronic mail notification of the 
new annotation- 
Regarding claims 11-12, presenting, as part of the user interface, a default set of recipients to 
receive the electronic mail notification, Hou: col 4/lines 26-31). 

Regarding claims 17, receiving an electronic mail notification of a new annotation 
coiresponding to media content (Biirel: receiving announcement, notice, indication, hint, i.e. 
notification that an electronic mail message has been received, col 1 1/lines 43-52, received 
electronic mail notifying new annotation coiresponding to one of a plurality of temporal 
segments after the beginning of the media content (Hou: temporal beginning/end marker, Figs. 6, 
14-15, col 6/line 10-14, col 2/line 52-61); 

displaying the electronic mail notification to the user (Biirel: col 1 1/lines 43-52, sending 
and displaying mail: col 1 1/lines 22-25); 

user selecting an identifier presented by the electronic mail notification of the media 
content and accessing responsive to said selection (col 12/lines 1-1 1, col 1 2/line 61 -col 13/hne 7) 
and rendered media content coiresponding to said temporal segment (Sidana: displaying col 
4/lines 14-43, accessing via user input: col 5/lines 58-col 6/line 6, col 7/lines 62-col 8/line 26, 
wherein user input comprises the selection of a URL of the media content identifier, col 12/lines 
2-27 used to retrieve multimedia content at location associated with the identifier upon user 
selection of the media content associated with said annotation, wherein the said annotation does 
not include the media content created by the user, col 12/lines 59-col 13/line 13); 
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Regarding claim 18, displaying annotation content for new annotation to the user (Hou: col 
6/lines 38-56). 

Regarding claim 19, displaying annotation content for the new annotation (Hou: col 6/lines 38- 
56), including one identifier to the user including one content identifier that identifies the media 
content such as a range identifier that identifies a temporal range of the segment coiresponding 
to the new annotation (Sidana: in the electronic mail message, an identifier of the new 
annotation, col 8/lines 5-12, an identifier of the temporal range of media content Hou: col 8/lines 
61-63, col 9/lines 1-4, col 6/lines 61-66, col 7/lines 1-3), 

an annotation identifier that identifies the new annotation one of annotation sets that the 
new annotation is part of (Hou: col 5/lines 10-28). 

Regarding claim 20-21 user selection of a content identifier consisting of an Url (Biirell: 12/lines 
61 -col 13/line 7, identifier (hot-link) of media content to which. the annotation content 
coiresponds). 

Regarding claim 22, receiving a user input comprises receiving a user selection of an identifier of 
the annotation (Hou: col 6/lines 61-66, col 7/line 1-3). 

Regarding claim 23, further discusses the apparatus for receiving step discusses on claim 17 
apparatus, rejected for obviousness under U.S.C. 103, this same rationale is also applied to 
software implementation claim, claimed in terms of function, property or characteristic. 

Regarding claim 24, further user-selectable identifier comprises Url that identifies a server and a 
location at the server where the media content is located (Sidana: col 7/lines 62-col 8/line 26, 
BiiTel, location-URL, col 3/lines 44-55, links in email, col 12/lines 22-67, not the multimedia 
content, col 1 3/lines 1-7) 

Regarding claim 25, further discusses the apparatus for receiving step discusses on claim 17, 
apparatus for generating email communicated on claim 6, user selection discussed on claim 23, 
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rejected for obviousness under U.S.C. 103, this same rationale is also applied to software 
implementation claim, claimed in teims of function, property or characteristic. 

Claim 26, cancelled. 

Regarding claim 27, the electronic mail message further includes a unique identifier of the new 
annotation (Sidana: col 7/lines 62-col 8/line 26). 

Regarding claim 28, the electronic mail message fiarther includes an identifier of one of a 
plurality of annotation sets, coiresponding to categories (e.g. foi*mat types) for annotations, that 
the new annotation is associated with (Hou: col 2/lines 52-61, e.g. voice, etc. col 6/lines 10-14). 

Regarding claims 29-30, annotation server, client computer running an user interface (Sidana: 
user interface to be presented with annotations, to the client computer abstract, annotation server 
120, col 8/lines 5-12); 

Regarding claim 33, receiving an electronic mail thread including a plurality of electronic mail 
messages (Biirell: news group annotations (e.g. notes), col 9/line 31-39 organizing received mail 
thread); creating a plurality of annotation fi*om at least one of the electronic mail messages in the 
electronic mail thread (Hou: generating an electronic mail message including both the content of 
the new annotation and an identifier of the media content, create electronic mail messages col 
3/lines 8-13, create means: col 2/lines 25-37, col 3/lines 9-13); and adding a plurality of 
annotation to an annotation database (Hou: add the new annotations, col 9/lines 15-32, col 1/lines 
17-20, adding annotations means: col 5/lines 29-52, report: col 6/lines 10-14, Fig. 3, saving in 
annotation database). 

Claim 34, cancelled. 

Regarding claim 35, the creating comprises generating, for each of the plurality of electronic 
mail messages, an annotation (Hou: generating an electronic mail message including both the 
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content of the new annotation and an identifier of the media content, create electronic mail 
messages col 3/lines 8-13, create means: col 2/lines 25-37, col 3/lines 9-13); and the adding 
comprises adding each of the generated annotations to the annotation database (Hou: add the new 
annotations, col 9/lines 15-32, col 1/lines 17-20, adding annotations means: col 5/lines 29-52, 
report: col 6/lines 10-14, Fig. 3, saving in annotation database). 

Regarding claim 36, locating, in the electronic mail thread, an identifier of media content that the 
annotation coiresponds to (Sidana: mail message further includes a unique identifier of the new 
annotation, col 7/lines 62-col 8/line 26, wherein the electronic mail message further includes an 
identifier of one or more annotation sets that the new annotation is associated with, Hou: col 
5/lines 10-28, BiiTell: mail thread, e.g. news groups annotations (e.g. notes), col 9/line 31-39 
organizing received mail thread). 

Regarding claims 40, link to installation progi'am (Biirel: col 1 2/lines 35-44). 

Regarding claims 41-42, substantially the same limitations as discussed on claims 1,17, and 23 
and further an field including data identifying a sender of the electronic mail message as an 
author of the new annotation coiTesponding to the media content included in the electronic mail 
message (Biirel, col 11 /lines 53-67); and 

a field including data identifying the time at which the new annotation is generated, a 
field including data identifying a title of the new annotation (Biirel, col 1 2/lines 12-14, a field 
including the located data (Biirel, col 1 2/lines 1-11, and a field including data identifying the 
media content to which the new annotation coiresponds (Biirel, col 1 2/lines 22-44); 
creation/time (Sidana: col 7/lines 1-6), title (Sidana: col 8/lines 44-53). 

5. Claims 13-16, are rejected under 35 U.S.C. 103(a) as being unpatentable over Hou- 
Sidana in view of Biirel U.S. Patent No. 6,009,462 in further view of Russell et. al (Russell) 
U.S. Patent No. 5,526,407. 
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Regarding claim 13, substantially the same as claims 1 and 15, discussed above, further, however 
the above combined teachings do not explicitly teach where an installation option that identifies 
an installation program that can be used to install one modules allowing the content and 
coiresponding media content to be rendered; 

Russell teaches a system/method related to for capturing, categorizing, and organizing by 
categories by using temporal segment identifiers of the media content, managing and retrieving 
multimedia (combined integrated media: audio, video text, graphics, etc. information) (see 
Russell: col 1/line 22-col 2/line 36, teaching means for annotating text to voice media content, 
col 3/lines 3-26, col 5/line 57-col 6/line 36, 45-col 7/line 16, identifiers of temporal segments of 
the multimedia content that are after the beginning of the multimedia content: col 10/line 60-col 
1 1/line 34, col 11/line 65-col 12/line 16, col 13/lines 3-31, 57-64), disclosing installation option 
that identifies an installation program that can be used to install one module allowing the 
content and coiresponding media content to be rendered (col 16/lines 47-col 19/line 35). 

It would have been obvious to one ordinary skilled in the art at the time the invention was 
made to incoiporated extendable or optional functionality allowing the content media 
type/format to be displayed (rendered), as taught by Russell, motivation would be enhance 
existing annotation/playback-based system with of replaying annotated multimedia content in a 
significantly order that which the media was originally stored, where temporal markers enable 
the support the enhancement of a playback that will require less time to render media content and 
enables the user to navigate through the media content by using visual cues. 

Regarding claim 14, including in the electronic mail message an identifier of the new annotation 
(Sidana: col 8/lines 5-12). 

Regarding claim 15, including in the electronic mail message, an identifier of the temporal range 
of media content (Hou: email including multimedia and attachments, col 1 /lines 39-44, 
multimedia attachments, col 6/lines 4-25, annotation event-related included the annotation file, 
col 7/lines 1-3, recording generates time-slice events, col 7/lines 4-6, time-lime markers include 
markers to specify annotation segments, col 8/lines 43-48, 61-63, col 9/lines 8-13). 
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Regarding claim 16, method/progi*am (Sidana: col 1/lines 55-62). 

Related U.S. Patents: 

The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure; pertinence is presented in accordance with to MPEP§ 707.05, Copies of documents 
cited will be provided as set forth in MPEP§ 707.05(a): 

Ref A: Integrated multimedia messages concepts and applications, Symposiums on Applied 
Computing, Proceedings of the 1996, ACM, ISBN: 0-89791-820-7, 1996, Philadelphia, 
Pennsylvania, pages 12-16. 

Pizano et. al. teaches extracting multimedia content fi*om and email message and storing 
retrieved content elsewhere in the network, using external-references or pointers provided to the 
recipient which is to retrieved stored multimedia content upon request. 
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Any inquiry concerning this communication or earlier communications fi'om the examiner 
should be directed to Prieto, B. whose telephone number is (703) 305-0750. The Examiner can 
normally be reached on Monday-Friday fi*om 6:00 to 3:30 p.m. If attempts to reach the examiner 
by telephone are unsuccessful, the Examiner*s Supervisor, Mark H. Rinehart can be reached on 
(703) 305-4815. The fax phone number for the organization where this application or 
proceeding is assigned is (703) 308-6606. Any inquiry of a general nature or relating to the status 
of this application or proceeding should be directed to the receptionist whose telephone number 
is (703) 305-3800/4700. 
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Washington, D.C. 20231 
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Or: 
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